Individualized hotword detection models

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for presenting notifications in an enterprise system. In one aspect, a method include actions of obtaining enrollment acoustic data representing an enrollment utterance spoken by a user, obtaining a set of candidate acoustic data representing utterances spoken by other users, determining, for each candidate acoustic data of the set of candidate acoustic data, a similarity score that represents a similarity between the enrollment acoustic data and the candidate acoustic data, selecting a subset of candidate acoustic data from the set of candidate acoustic data based at least on the similarity scores, generating a detection model based on the subset of candidate acoustic data, and providing the detection model for use in detecting an utterance spoken by the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/197,268, filed on Jun. 29, 2016, which is a continuation of U.S.application Ser. No. 14/805,753, filed on Jul. 22, 2015. The disclosuresof the prior applications are considered part of and are incorporated byreference in the disclosure of this application.

TECHNICAL FIELD

This disclosure generally relates to controlling computers using voicecommands.

BACKGROUND

A computer may analyze a user's utterance and may perform an action inresponse. For example, a user may say “DRIVE HOME” and a computer mayrespond with directions for the user to drive home from their currentlocation.

SUMMARY

In general, an aspect of the subject matter described in thisspecification may involve a process for generating an individualizedhotword detection model. As used by this specification, a “hotword” mayrefer to a term that wakes a device up from a sleep state or hibernationstate, or a term that triggers semantic interpretation on the term or onone or more terms that follow the term, e.g., on voice commands thatfollow the hotword.

For example, in the utterance “OK COMPUTER, DRIVE HOME,” the term “OKCOMPUTER,” may be a hotword that triggers semantic interpretation on thefollowing term “DRIVE HOME,” and the term “DRIVE HOME” may correspond toa voice command for providing directions to the user's home. When thesystem receives sound corresponding to the utterance “OK COMPUTER, DRIVEHOME,” the system may determine that the utterance begins with thehotword, “OK COMPUTER,” in response, transcribe the sound, performsemantic interpretation on the transcription of the voice command “DRIVEHOME,” and output directions for the user to drive home.

Hotwords may be useful for “always on” systems that may potentially pickup sounds that are not directed to the system. For example, the use ofhotwords may help the system discern when a given utterance is directedat the system, as opposed to an utterance that is directed to anotherindividual present in the environment. In doing so, the system may avoidcomputationally expensive processing, e.g., semantic interpretation, onsounds or utterances that do not include a hotword.

A system may detect an utterance includes a hotword based on a hotworddetection model. However, different users may pronounce the same hotwordin different ways. Accordingly, the system may not detect when someusers speak the hotword. The system may increase detection of hotwordsbased on generating individualized hotword detection models. However,generating a hotword detection model may use thousands of utterances anda user may not desire to provide thousands of enrollment utterances.Accordingly, after receiving one or more enrollment utterances by auser, the system may identify other utterances of the hotword by otherusers, select the utterances that are similar to the enrollmentutterances by the user, and generate the individualized hotworddetection model using the selected utterances and the enrollmentutterances.

In some aspects, the subject matter described in this specification maybe embodied in methods that may include the actions of obtainingenrollment acoustic data representing an enrollment utterance spoken bya user, obtaining a set of candidate acoustic data representingutterances spoken by other users, determining, for each candidateacoustic data of the set of candidate acoustic data, a similarity scorethat represents a similarity between the enrollment acoustic data andthe candidate acoustic data, selecting a subset of candidate acousticdata from the set of candidate acoustic data based at least on thesimilarity scores, generating a detection model based on the subset ofcandidate acoustic data, and providing the detection model for use indetecting an utterance spoken by the user.

Other versions include corresponding systems, apparatus, and computerprograms, configured to perform the actions of the methods, encoded oncomputer storage devices.

These and other versions may each optionally include one or more of thefollowing features. For instance, in some implementations obtainingenrollment acoustic data representing an enrollment utterance spoken bya user includes obtaining enrollment acoustic data for multipleutterances of a predetermined phrase spoken by the user.

In certain aspects, obtaining a set of candidate acoustic datarepresenting utterances spoken by other users includes determining theenrollment utterance is of a predetermined phrase and identifyingcandidate acoustic data representing utterances of the predeterminedphrase spoken by other users.

In some aspects, determining, for each candidate acoustic data of theset of candidate acoustic data, the similarity score includesdetermining a distance between the enrollment acoustic data and thecandidate acoustic data and determining the similarity score based onthe distance.

In some implementations, determining, for each candidate acoustic dataof the set of candidate acoustic data, the similarity score includesdetermining the similarity scores based on demographic information ofthe other user that spoke the utterance represented by the candidateacoustic data and demographic information of the user that spoke theenrollment utterance.

In certain aspects, selecting a subset of candidate acoustic data fromthe set of candidate acoustic data based at least on similarity scoresof the candidate acoustic data that represent a similarity between theenrollment acoustic data and the candidate acoustic data is based onselecting a predetermined number of candidate acoustic data.

In some aspects, generating a detection model based on the subset ofcandidate acoustic data includes training a neural network using thesubset of candidate acoustic data. In some implementations, additionalactions include detecting an utterance of a predetermined phrase usingthe detection model.

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other potential features, aspects,and advantages of the subject matter will become apparent from thedescription, the drawings, and the claims.

DESCRIPTION OF DRAWINGS

FIGS. 1 and 2 are illustrations of block diagrams of example systems forgenerating an individualized hotword detection model.

FIG. 3 is a flowchart of an example process for generating anindividualized hotword detection model.

FIG. 4 is a diagram of exemplary computing devices.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of an example system 100 for generating anindividualized hotword detection model. Briefly, and as described infurther detail below, the system 100 may include a client device 120 anda server 130 that includes a candidate acoustic data scorer 134,candidate acoustic data selector 136, and a hotword detection modelgenerator 140.

The client device 120 may be a smart phone, a laptop computer, a tabletcomputer, a desktop computer, or some other computing device that isconfigured to detect when a user 110 says a hotword. For example, theclient device 120 may be configured to detect when the user 110 says “OKCOMPUTER.”

The client device 120 may detect when the user 110 speaks a hotwordusing a hotword detection model. For example, the client device 120 maydetect a user is speaking “OK COMPUTER” using a hotword detection modelthat has been trained to detect sounds corresponding to when thehotword, “OK COMPUTER,” is spoken.

However, different users may pronounce the same hotword in differentways. For example, the user 110 may pronounce “OK COMPUTER” as “OKCOM-UT-ER,” and a hotword detection model may not detect “OK COM-UT-ER”as “OK COMPUTER.” Accordingly, the client device 120 may increasedetection of hotwords spoken by the user 110 based on a personalizedhotword detection model 152 that is trained to detect when the user 110says the hotword. For example, the personalized hotword detection model152 may be trained to detect “OK COM-UT-ER” as a user's 110pronunciation of the hotword “OK COMPUTER.”

To obtain the personalized hotword detection model 152, the clientdevice 120 may prompt the user to provide enrollment utterances. Forexample, for obtaining a personalized hotword detection model fordetecting the hotword “OK COMPUTER,” the client device 120 may providethe prompt “NOW PERSONALIZING HOTWORD DETECTION, SAY ‘OK COMPUTER’ THREETIMES” to the user 110. The client device 120 may include an acousticdata generator that captures sound as acoustic data. For example, theclient device 120 may include a microphone that captures the user 110speaking “OK COMPUTER” as “OK COM-UT-ER” as signals, and encodes thesignals as enrollment acoustic data 122 represented by mel-frequencycepstral coefficients.

The client device 120 may provide the enrollment acoustic data 122 to aserver 130 and in response receive the personalized hotword detectionmodel 152. For example, the client device 120 may provide enrollmentacoustic data 122 representing the user 110 speaking “OK COMPUTER” as“OK COM-UT-ER” to the server 130, and in response, receive thepersonalized hotword detection model 152 trained based at least on theenrollment acoustic data.

The client device 120 may then detect when the user speaks the hotwordusing the personalized hotword detection model 152. For example, usingthe personalized hotword detection model 152 trained based on theenrollment acoustic data 122 representing the user 110 speaking “OKCOMPUTER” as “OK COM-UT-ER,” the client device 120 may detect the user110 is saying the hotword “OK COMPUTER” when the user says “OKCOM-UT-ER.”

The server 130 may be configured to generate a personalized hotworddetection model based on enrollment acoustic data. For example, theserver 130 may receive the enrollment acoustic data 122 representing theuser 110 speaking “OK COMPUTER” as “OK COM-UT-ER” and train thepersonalized hotword detection model 152 based at least on theenrollment acoustic data.

However, generating a hotword detection model may use thousands ofutterances and a user may not want to personally provide thousands ofenrollment utterances. Accordingly, after receiving one or moreenrollment utterances by a user, the server 130 may identify otherutterances of the hotword by other users, select the utterances that aresimilar to the enrollment utterances by the user, and generate thepersonalized hotword detection model 152 using the selected utterancesand the enrollment utterances.

In more detail, the candidate acoustic database 132 of the server 130may store acoustic data representing utterances of various users. Forexample, the candidate acoustic database 132 of the server 130 may storeacoustic data representing hundreds of thousands of utterances ofdifferent users. The candidate acoustic database 132 may store eachacoustic data with data that indicates the hotword that was uttered. Forexample, the candidate acoustic database 132 may store fifty thousandsets of acoustic data labeled as being an utterance of the hotword “OKCOMPUTER” and fifty thousand sets of acoustic data labeled as being anutterance of a different hotword “MY BUTLER.” In some implementations,the candidate acoustic database 132 may associate the acoustic data withdemographic data that describes a user. For example, the candidateacoustic database 132 may associate the acoustic data with a locationthat the user was in when the hotword was spoken by the user. In anotherexample, the candidate acoustic database 132 may associate the acousticdata with a gender of the user, an age range of the user, or some otherinformation describing the user.

The candidate acoustic data scorer 134 of the server 130 may beconfigured to obtain the enrollment acoustic data 122 and the candidateacoustic data from the candidate acoustic database 132 and generate asimilarity score that represents a similarity between the enrollmentacoustic data 122 and the candidate acoustic data. For example, thecandidate acoustic data scorer 134 may receive enrollment acoustic data122 of the user saying “OK COMPUTER” and candidate acoustic datarepresenting another user saying “OK COMPUTER,” determine a 90%similarity, and associate a score of 0.9 with the candidate acousticdata. In the example, the candidate acoustic data scorer 134 may thenobtain a second set of candidate acoustic data representing yet anotheruser saying “OK COMPUTER,” determine a 30% similarity with theenrollment acoustic data 122, and associate a score of 0.3 with thesecond set of candidate acoustic data.

The similarity score of a candidate acoustic data representing aparticular utterance may reflect an acoustic similarity between theparticular utterance and an enrollment utterance. For example, thesimilarity score may range from 0 to 1 where higher similarity scoresreflect greater acoustic similarity and lower scores reflect loweracoustic similarity. In other examples other types of scores and rangesmay be used, e.g., 1-5, A-F, or 0%-100%.

The candidate acoustic data scorer 134 may generate the score based on adistance between the enrollment acoustic data and the candidate acousticdata. For example, the candidate acoustic data scorer 134 may aggregatea difference between mel-frequency cepstral coefficients of theenrollment acoustic data and the candidate acoustic data across multipleframes, and determine a similarity score where greater aggregatedistances result in scores that reflect less similarity and loweraggregate distances result in scores that reflect more similarity.

In some implementations, the candidate acoustic data scorer 134 maydetermine the score based on demographic information of the other user.For example, instead of selecting candidate acoustic data representingutterances of a user with the same gender, the candidate acoustic datascorer 134 may obtain candidate acoustic data representing utterances ofusers of different genders, determine whether the gender of a userspeaking the utterance represented by the candidate acoustic datamatches the gender of the user 110, and in response to determining amatch, assigning a higher similarity score to candidate acoustic datarepresenting utterances of users of the same gender as the user 110.

In some implementations, the candidate acoustic data scorer 134 mayselect candidate acoustic data from among more candidate acoustic datastored in the candidate acoustic database 132. For example, thecandidate acoustic data scorer 134 may select to receive the acousticdata from the candidate acoustic database 132 where the hotword “OKCOMPUTER” is spoken. The candidate acoustic data scorer 134 may obtain,with the enrollment acoustic data, one or more of an indication of thehotword spoken or an indication of the type of user saying the hotword,and query the candidate acoustic database 132 for acoustic data of userssaying the same hotword or a similar type of user to the user saying thehotword. For example, the candidate acoustic data scorer 134 may obtainan indication that the hotword “OK COMPUTER” was spoken by a femaleuser, and in response, query the candidate acoustic database 132 foracoustic data representing the hotword “OK COMPUTER” being spoken by afemale user.

The candidate acoustic data selector 136 may obtain the scored candidateacoustic data from the candidate acoustic data scorer 134 and theenrollment acoustic data 122, and generate a training set 138 ofacoustic data for training the personalized hotword detection model 152.For example, the candidate acoustic data selector 136 may obtainenrollment acoustic data representing the user 110 speaking “OKCOMPUTER” and obtain fifty thousand of candidate acoustic datarepresenting different other users saying “OK COMPUTER,” where each ofthe candidate acoustic data is associated with a similarity scorereflecting a similarity between the candidate acoustic data and theenrollment acoustic data 122, and generate a training set of acousticdata including ten thousand of the fifty thousand candidate acousticdata and the enrollment acoustic data 122.

The candidate acoustic data selector 136 may generate the training set138 based on selecting a subset of the candidate acoustic data based atleast on the similarity scores. For example, the candidate acoustic dataselector 136 may obtain a set of fifty thousand candidate acoustic dataand select a subset of ten thousand candidate acoustic data of the setwith similarity scores that reflect higher similarities between thecandidate acoustic data and the enrollment acoustic data 122 than theother candidate acoustic data.

The candidate acoustic data selector 136 may select the subset ofcandidate acoustic data based on selecting a predetermined number, e.g.,one thousand, three thousand, ten thousand, fifty thousand, of candidateacoustic data. For example, the candidate acoustic data selector 136 mayobtain enrollment acoustic data representing a single utterance of “OKCOMPUTER,” and select a subset of three thousand candidate acoustic datawith similarity scores that reflect a higher similarity between thecandidate acoustic data and the enrollment acoustic data.

Additionally or alternatively, the candidate acoustic data selector 136may select a subset of candidate acoustic data based on selectingcandidate acoustic data that satisfies a threshold similarity score. Forexample, the candidate acoustic data selector 136 may select candidateacoustic data with similarity scores above a threshold similarity scoreof 0.8, 0.85, 0.9 from a score range of 0.0-1.0, and include theselected candidate acoustic data in the training set 138.

In some implementations, the candidate acoustic data selector 136 mayweight the acoustic data in the training set 138. For example, thecandidate acoustic data selector 136 may include an enrollment acousticdata multiple times in the training set 138 or associate the enrollmentacoustic data in the training set 138 with a greater weight thancandidate acoustic data.

In some implementations, the candidate acoustic data selector 136 mayselect the subset of candidate acoustic data based on multipleenrollment acoustic data. For example, the candidate acoustic dataselector 136 may receive enrollment acoustic data for three utterancesof “OK COMPUTER” by the user 110, and for each enrollment acoustic data,select three thousand of the candidate acoustic data with similarityscores that reflect the most similarity to include in the training set138. Accordingly, some candidate acoustic data may appear in thetraining set 138 multiple times if the candidate acoustic data isselected for multiple enrollment acoustic data. In some implementations,the candidate acoustic data selector 136 may remove duplicate candidateacoustic data from the training set 138 or prevent duplicate candidateacoustic data from being included in the training set 138.

In some implementations, the candidate acoustic data selector 136 maydetermine the number of candidate acoustic data to select for anenrollment acoustic data based on a number of enrollment acoustic datareceived by the candidate acoustic data selector 136. For example, thecandidate acoustic data selector 136 may receive five enrollmentacoustic data, determine that the hotword detection model generatorshould receive at a training set of least ten thousand acoustic data,and in response, for each enrollment acoustic data received, select atleast one thousand nine hundred ninety-nine candidate acoustic data toinclude in the training set with the enrollment acoustic data. Inanother example, the candidate acoustic data selector 136 may receiveten enrollment acoustic data, determine that the hotword detection modelgenerator should receive at a training set of least ten thousandacoustic data, and in response, for each enrollment acoustic datareceived, select at least nine hundred ninety nine candidate acousticdata to include in the training set with the enrollment acoustic data.

In another example, the candidate acoustic data selector 136 maydetermine a similarity score for the candidate acoustic data based ondetermining sub-similarity scores for each of multiple enrollmentacoustic data. For example, the candidate acoustic data selector 136 mayreceive three enrollment acoustic data, and for each candidate acousticdata, determine three sub-similarity scores each corresponding to one ofthe enrollment acoustic data, and determine the similarity score basedon averaging the sub-similarity scores. In yet another example, thecandidate acoustic data selector may take a median, floor, or ceiling ofsub-similarity scores for a candidate acoustic data as the similarityscore.

The hotword detection model generator 140 may receive the training set138 from the candidate acoustic data selector 136 and generate apersonalized hotword detection model 152. For example, the hotworddetection model generator 140 may receive a training set including ninethousand nine hundred and ninety-seven selected candidate acoustic dataand three enrollment acoustic data, and generate a personalized hotworddetection model 152 based on the training set.

The hotword detection model generator 140 may generate the personalizedhotword detection model 152 based on training a neural network to detectthe acoustic data in the training set 138 as representing utterances ofthe hotword. For example, the hotword detection model generator 140 maygenerate the personalized hotword detection model 152 that detects thehotword “OK COMPUTER” based on the acoustic data in the training set138.

Different configurations of the system 100 may be used wherefunctionality of the client device 120 and the server 130 that includesthe candidate acoustic data scorer 134, the candidate acoustic dataselector 136, and the hotword detection model generator 140 may becombined, further separated, distributed, or interchanged. The system100 may be implemented in a single device or distributed across multipledevices.

FIG. 2 is a block diagram of example server 130 for generating anindividualized hotword detection model. The server 130 may be the serverdescribed in FIG. 1. As described above, the server 130 may include thecandidate acoustic database 132, the candidate acoustic data scorer 134,the candidate acoustic data selector 136, and the hotword detectionmodel generator 140.

The candidate acoustic database 132 may include multiple candidateacoustic data of various users saying the hotword “OK COMPUTER.” Forexample, the candidate acoustic database 132 may include a candidateacoustic data of “User A” saying “OK COMPUTER” as “OK COM-PU-TER,” acandidate acoustic data of “User B” saying “OK COMPUTER” as “OOKCOM-PU-TER”, a candidate acoustic data of “User C” saying “OK COMPUTER”as “OK COP-TER,” a candidate acoustic data of “User D” saying “OKCOMPUTER” as “OK COM-U-TER,” a candidate acoustic data of “User E”saying “OK COMPUTER” as “OK COM-MUT-ER,” a candidate acoustic data of“User F” saying “OK COMPUTER” as “OK COM-PUT-EW,” and other candidateacoustic data of other users saying “OK COMPUTER.”

The candidate acoustic data scorer 134 may receive enrollment acousticdata 202 of a user and obtain a set of candidate acoustic data from thecandidate acoustic database 132. For example, the candidate acousticdata scorer 134 may receive enrollment acoustic data 202 of the usersaying “OK COMPUTER” as “OK COM-UT-ER,” and in response, obtain a set ofcandidate acoustic data from the candidate acoustic database 132including the candidate acoustic data of “User A” saying “OK COMPUTER”as “OK COM-PU-TER,” the candidate acoustic data of “User B” saying “OKCOMPUTER” as “OOK COM-PU-TER,” the candidate acoustic data of “User C”saying “OK COMPUTER” as “OK COP-TER,” the candidate acoustic data of“User D” saying “OK COMPUTER” as “OK COM-U-TER,” the candidate acousticdata of “User E” saying “OK COMPUTER” as “OK COM-MUT-ER,” the candidateacoustic data of “User F” saying “OK COMPUTER” as “OK COM-PUT-EW,” andthe other candidate acoustic data of other users saying “OK COMPUTER.”

The candidate acoustic data scorer 134 may generate similarity scoresfor each of the set of candidate acoustic data. For example, for anenrollment acoustic data of the user 110 saying “OK COMPUTER” as “OKCOM-UT-ER,” the candidate acoustic data scorer 134 may generate asimilarity score of 0.6 reflecting a moderate similarity for candidateacoustic data of “User A” saying “OK COMPUTER” as “OK COM-PU-TER,” asimilarity score of 0.5 reflecting a moderate similarity for candidateacoustic data of “User B” saying “OK COMPUTER” as “OOK COM-PU-TER”, asimilarity score of 0.3 reflecting a low similarity for candidateacoustic data of “User C” saying “OK COMPUTER” as “OK COP-TER,” asimilarity score of 0.9 reflecting a high similarity for candidateacoustic data of “User D” saying “OK COMPUTER” as “OK COM-U-TER,” asimilarity score of 0.8 reflecting a high similarity for candidateacoustic data of “User E” saying “OK COMPUTER” as “OK COM-MUT-ER,” and asimilarity score of 0.5 reflecting a moderate similarity for candidateacoustic data of “User F” saying “OK COMPUTER” as “OK COM-PUT-EW.”

The candidate acoustic data selector 136 may receive the scoredcandidate acoustic data 204 from the candidate acoustic data scorer 134and generate the training set 138 of acoustic data. For example, thecandidate acoustic data selector 136 may receive a similarity score of0.6 reflecting a moderate similarity for candidate acoustic data of“User A” saying “OK COMPUTER” as “OK COM-PU-TER,” a similarity score of0.5 reflecting a moderate similarity for candidate acoustic data of“User B” saying “OK COMPUTER” as “OOK COM-PU-TER”, a similarity score of0.3 reflecting a low similarity for candidate acoustic data of “User C”saying “OK COMPUTER” as “OK COP-TER,” a similarity score of 0.9reflecting a high similarity for candidate acoustic data of “User D”saying “OK COMPUTER” as “OK COM-U-TER,” a similarity score of 0.8reflecting a high similarity for candidate acoustic data of “User E”saying “OK COMPUTER” as “OK COM-MUT-ER,” a similarity score of 0.5reflecting a moderate similarity for candidate acoustic data of “User F”saying “OK COMPUTER” as “OK COM-PUT-EW,” the corresponding candidateacoustic data, and the enrollment acoustic data, and in response maygenerate a training set of acoustic data including the candidateacoustic data of “User D” saying “OK COMPUTER” as “OK COM-U-TER,” thecandidate acoustic data of “User E” saying “OK COMPUTER” as “OKCOM-MUT-ER,” and the enrollment acoustic data of the user saying “OKCOMPUTER” as “OK COM-UT-ER.”

The candidate acoustic data selector 136 may generate the training setby selecting a subset of the set of candidate acoustic data based on thesimilarity scores. For example, the candidate acoustic data selector 136may determine that the hotword detection model generator should receivea training set of three acoustic data, determine there is one enrollmentacoustic data, determine to select two candidate acoustic data to obtainthree total acoustic data, and select the candidate acoustic data withthe similarity scores of 0.9 and 0.8 that reflect the greatestsimilarity with the enrollment acoustic data out of all of the candidateacoustic data.

The hotword detection model generator 140 may receive the training set138 and generate a personalized hotword detection model 152. Forexample, the hotword detection model generator 140 may receive atraining set including the candidate acoustic data of “User D” saying“OK COMPUTER” as “OK COM-U-TER,” the candidate acoustic data of “User E”saying “OK COMPUTER” as “OK COM-MUT-ER,” and the enrollment acousticdata of the user saying “OK COMPUTER” as “OK COM-UT-ER,” and train aneural network to detect those acoustic data as representing the hotword“OK COMPUTER” being spoken by the user 110.

FIG. 3 is a flowchart of an example process for generating anindividualized hotword detection model. The following describes theprocessing 300 as being performed by components of the systems 100 thatare described with reference to FIG. 1. However, the process 300 may beperformed by other systems or system configurations.

The process 300 may include obtaining enrollment acoustic datarepresenting an enrollment utterance spoken by a user (310). Forexample, the candidate acoustic data scorer 134 may obtain enrollmentacoustic data from the client device 120 representing the user saying ahotword, “MY BUTLER,” after being prompted by the client device 120 toprovide a sample enrollment utterance for training the client device 120to detect when the user says the hotword, “MY BUTLER.”

The process 300 may include obtaining a set of candidate acoustic datarepresenting utterances spoken by other users (320). For example, thecandidate acoustic data scorer 134 may determine that the enrollmentacoustic data is for the hotword, “MY BUTLER,” spoken by a male betweenthe ages of twenty to thirty, and in response, obtain, from thecandidate acoustic database 132, candidate acoustic data representingother male users between the ages of twenty to thirty saying thehotword, “MY BUTLER.”

The process may include determining, for each candidate acoustic data ofthe set of candidate acoustic data, a similarity score that represents asimilarity between the enrollment acoustic data and the candidateacoustic data (330). For example, for each candidate acoustic dataobtained from the candidate acoustic database 132, the candidateacoustic data scorer 134 may determine enrollment acoustic datarepresenting the user saying the hotword, “MY BUTLER,” and the candidateacoustic data representing another user saying the hotword, “MY BUTLER.”

The process may include selecting a subset of candidate acoustic datafrom the set of candidate acoustic data based at least on the similarityscores (340). For example, the candidate acoustic data selector 136 mayselect a predetermined number, e.g., one thousand, five thousand, twentythousand, or some other number, of candidate acoustic data with thesimilarity scores that reflect the most similarity with the enrollmentacoustic data. In another example, the candidate acoustic data selector136 may select candidate acoustic data with similarity scores thatsatisfy a threshold similarity score, e.g., 0.7, 0.8, 0.9, or some otheramount.

The process may include generating a detection model based on the subsetof candidate acoustic data (350). For example, the hotword detectionmodel generator 140 may generate the personalized hotword detectionmodel based on training a neural network to detect when the user speaksthe hotword, “MY BUTLER,” using the selected candidate acoustic data ofother users saying “MY BUTLER.”

The process may include providing the detection model for use indetecting an utterance spoken by the user (360). For example, the server130 may provide the personalized hotword detection model 152 generatedby the hotword detection model generator 140 to the client device 120.The client device 120 may then use the personalized hotword detectionmodel 152 for detecting when the user 110 says the hotword, “MY BUTLER.”

FIG. 4 shows an example of a computing device 400 and a mobile computingdevice 450 that can be used to implement the techniques described here.The computing device 400 is intended to represent various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. The mobile computing device 450 is intended torepresent various forms of mobile computing devices, such as personaldigital assistants, cellular telephones, smart-phones, and other similarcomputing devices. The components shown here, their connections andrelationships, and their functions, are meant to be examples only, andare not meant to be limiting.

The computing device 400 includes a processor 402, a memory 404, astorage device 406, a high-speed interface 408 connecting to the memory404 and multiple high-speed expansion ports 410, and a low-speedinterface 412 connecting to a low-speed expansion port 414 and thestorage device 406. Each of the processor 402, the memory 404, thestorage device 406, the high-speed interface 408, the high-speedexpansion ports 410, and the low-speed interface 412, are interconnectedusing various busses, and may be mounted on a common motherboard or inother manners as appropriate. The processor 402 can process instructionsfor execution within the computing device 400, including instructionsstored in the memory 404 or on the storage device 406 to displaygraphical information for a graphical user interface (GUI) on anexternal input/output device, such as a display 416 coupled to thehigh-speed interface 408. In other implementations, multiple processorsand/or multiple buses may be used, as appropriate, along with multiplememories and types of memory. Also, multiple computing devices may beconnected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

The memory 404 stores information within the computing device 400. Insome implementations, the memory 404 is a volatile memory unit or units.In some implementations, the memory 404 is a non-volatile memory unit orunits. The memory 404 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 406 is capable of providing mass storage for thecomputing device 400. In some implementations, the storage device 406may be or contain a computer-readable medium, such as a floppy diskdevice, a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. Instructions can be stored in an information carrier.The instructions, when executed by one or more processing devices (forexample, processor 402), perform one or more methods, such as thosedescribed above. The instructions can also be stored by one or morestorage devices such as computer- or machine-readable mediums (forexample, the memory 404, the storage device 406, or memory on theprocessor 402).

The high-speed interface 408 manages bandwidth-intensive operations forthe computing device 400, while the low-speed interface 412 manageslower bandwidth-intensive operations. Such allocation of functions is anexample only. In some implementations, the high-speed interface 408 iscoupled to the memory 404, the display 416 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 410,which may accept various expansion cards (not shown). In theimplementation, the low-speed interface 412 is coupled to the storagedevice 406 and the low-speed expansion port 414. The low-speed expansionport 414, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 400 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 420, or multiple times in a group of such servers. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 422. It may also be implemented as part of a rack server system424. Alternatively, components from the computing device 400 may becombined with other components in a mobile computing device (not shown),such as a mobile computing device 450. Each of such devices may containone or more of the computing device 400 and the mobile computing device450, and an entire system may be made up of multiple computing devicescommunicating with each other.

The mobile computing device 450 includes a processor 452, a memory 464,an input/output device such as a display 454, a communication interface466, and a transceiver 468, among other components. The mobile computingdevice 450 may also be provided with a storage device, such as amicro-drive or other device, to provide additional storage. Each of theprocessor 452, the memory 464, the display 454, the communicationinterface 466, and the transceiver 468, are interconnected using variousbuses, and several of the components may be mounted on a commonmotherboard or in other manners as appropriate.

The processor 452 can execute instructions within the mobile computingdevice 450, including instructions stored in the memory 464. Theprocessor 452 may be implemented as a chipset of chips that includeseparate and multiple analog and digital processors. The processor 452may provide, for example, for coordination of the other components ofthe mobile computing device 450, such as control of user interfaces,applications run by the mobile computing device 450, and wirelesscommunication by the mobile computing device 450.

The processor 452 may communicate with a user through a controlinterface 458 and a display interface 456 coupled to the display 454.The display 454 may be, for example, a TFT (Thin-Film-Transistor LiquidCrystal Display) display or an OLED (Organic Light Emitting Diode)display, or other appropriate display technology. The display interface456 may comprise appropriate circuitry for driving the display 454 topresent graphical and other information to a user. The control interface458 may receive commands from a user and convert them for submission tothe processor 452. In addition, an external interface 462 may providecommunication with the processor 452, so as to enable near areacommunication of the mobile computing device 450 with other devices. Theexternal interface 462 may provide, for example, for wired communicationin some implementations, or for wireless communication in otherimplementations, and multiple interfaces may also be used.

The memory 464 stores information within the mobile computing device450. The memory 464 can be implemented as one or more of acomputer-readable medium or media, a volatile memory unit or units, or anon-volatile memory unit or units. An expansion memory 474 may also beprovided and connected to the mobile computing device 450 through anexpansion interface 472, which may include, for example, a SIMM (SingleIn Line Memory Module) card interface. The expansion memory 474 mayprovide extra storage space for the mobile computing device 450, or mayalso store applications or other information for the mobile computingdevice 450. Specifically, the expansion memory 474 may includeinstructions to carry out or supplement the processes described above,and may include secure information also. Thus, for example, theexpansion memory 474 may be provided as a security module for the mobilecomputing device 450, and may be programmed with instructions thatpermit secure use of the mobile computing device 450. In addition,secure applications may be provided via the SIMM cards, along withadditional information, such as placing identifying information on theSIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory(non-volatile random access memory), as discussed below. In someimplementations, instructions are stored in an information carrier thatthe instructions, when executed by one or more processing devices (forexample, processor 452), perform one or more methods, such as thosedescribed above. The instructions can also be stored by one or morestorage devices, such as one or more computer- or machine-readablemediums (for example, the memory 464, the expansion memory 474, ormemory on the processor 452). In some implementations, the instructionscan be received in a propagated signal, for example, over thetransceiver 468 or the external interface 462.

The mobile computing device 450 may communicate wirelessly through thecommunication interface 466, which may include digital signal processingcircuitry where necessary. The communication interface 466 may providefor communications under various modes or protocols, such as GSM voicecalls (Global System for Mobile communications), SMS (Short MessageService), EMS (Enhanced Messaging Service), or MMS messaging (MultimediaMessaging Service), CDMA (code division multiple access), TDMA (timedivision multiple access), PDC (Personal Digital Cellular), WCDMA(Wideband Code Division Multiple Access), CDMA2000, or GPRS (GeneralPacket Radio Service), among others. Such communication may occur, forexample, through the transceiver 468 using a radio-frequency. Inaddition, short-range communication may occur, such as using aBluetooth, WiFi, or other such transceiver (not shown). In addition, aGPS (Global Positioning System) receiver module 470 may provideadditional navigation- and location-related wireless data to the mobilecomputing device 450, which may be used as appropriate by applicationsrunning on the mobile computing device 450.

The mobile computing device 450 may also communicate audibly using anaudio codec 460, which may receive spoken information from a user andconvert it to usable digital information. The audio codec 460 maylikewise generate audible sound for a user, such as through a speaker,e.g., in a handset of the mobile computing device 450. Such sound mayinclude sound from voice telephone calls, may include recorded sound(e.g., voice messages, music files, etc.) and may also include soundgenerated by applications operating on the mobile computing device 450.

The mobile computing device 450 may be implemented in a number ofdifferent forms, as shown in the figure. For example, it may beimplemented as a cellular telephone 480. It may also be implemented aspart of a smart-phone 482, personal digital assistant, or other similarmobile computing device.

Embodiments of the subject matter, the functional operations and theprocesses described in this specification can be implemented in digitalelectronic circuitry, in tangibly-embodied computer software orfirmware, in computer hardware, including the structures disclosed inthis specification and their structural equivalents, or in combinationsof one or more of them. Embodiments of the subject matter described inthis specification can be implemented as one or more computer programs,i.e., one or more modules of computer program instructions encoded on atangible nonvolatile program carrier for execution by, or to control theoperation of, data processing apparatus. Alternatively or in addition,the program instructions can be encoded on an artificially generatedpropagated signal, e.g., a machine-generated electrical, optical, orelectromagnetic signal that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. The computer storage medium can be amachine-readable storage device, a machine-readable storage substrate, arandom or serial access memory device, or a combination of one or moreof them.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them.

A computer program (which may also be referred to or described as aprogram, software, a software application, a module, a software module,a script, or code) can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astandalone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program may, butneed not, correspond to a file in a file system. A program can be storedin a portion of a file that holds other programs or data (e.g., one ormore scripts stored in a markup language document), in a single filededicated to the program in question, or in multiple coordinated files(e.g., files that store one or more modules, sub programs, or portionsof code). A computer program can be deployed to be executed on onecomputer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Computers suitable for the execution of a computer program include, byway of example, can be based on general or special purposemicroprocessors or both, or any other kind of central processing unit.Generally, a central processing unit will receive instructions and datafrom a read-only memory or a random access memory or both. The essentialelements of a computer are a central processing unit for performing orexecuting instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device (e.g., a universalserial bus (USB) flash drive), to name just a few.

Computer readable media suitable for storing computer programinstructions and data include all forms of nonvolatile memory, media andmemory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto optical disks; andCD-ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous. Other steps may be provided, or stepsmay be eliminated, from the described processes. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. (canceled)
 2. A computer-implemented methodcomprising: receiving audio data corresponding to a single utterance bya user of a predefined hotword, wherein the predefined hotword ispronounced by the user using a personalized, non-standard pronunciation;in response to receiving the audio data corresponding to the singleutterance by the user of the predefined hotword, downloading audiofeatures corresponding to other users' utterances of the same,predefined hotword in a manner that is indicated as similar to thepersonalized, non-standard pronunciation; dynamically generating ahotword detection model for the personalized, non-standard pronunciationof the predefined hotword using (i) the audio data corresponding to thesingle utterance by the user of the predefined hotword, and (ii) thedownloaded audio features corresponding to other users' utterances ofthe same, predefined hotword; and using the dynamically generatedhotword detection model to detect a likely utterance of the predefinedhotword in subsequently received audio data.
 3. The computer implementedmethod of claim 2, comprising: during an enrollment process, prompting,by a client device, the user to speak the predefined hotword; andgenerating enrollment acoustic data using the received audio data fromthe user, wherein the audio data comprises the predefined hotwordpronounced by the user using the personalized, non-standardpronunciation and additional one or more terms spoken by the user thattrigger semantic interpretation of the one or more terms that follow thepredefined hotword.
 4. The computer implemented method of claim 3,comprising: obtaining a set of candidate acoustic data representingutterances that were previously-spoken by the other users, wherein theother users are of a similar type of user to the user.
 5. The computerimplemented method of claim 4, wherein obtaining the set of candidateacoustic data representing the utterances that were spoken by the otherusers comprises: determining, for each candidate acoustic data of theset of candidate acoustic data, a similarity score that represents anacoustic similarity between the enrollment acoustic data and thecandidate acoustic data.
 6. The computer implemented method of claim 5,wherein determining the similarity score that represents an acousticsimilarity between the enrollment acoustic data and the candidateacoustic data comprises: determining a plurality of sub-similarityscores between the enrollment acoustic data and the candidate acousticdata; and determining the similarity score based on an averaging of theplurality of sub-similarity scores.
 7. The computer implemented methodof claim 2, wherein dynamically generating the hotword detection modelfor the personalized, non-standard pronunciation of the predefinedhotword using (i) the audio data corresponding to the single utteranceby the user of the predefined hotword, and (ii) the downloaded audiofeatures corresponding to other users' utterances of the same,predefined hotword comprises: training the hotword detection model todetect the likely utterance of the predefined hotword by the user in thesubsequently received audio data corresponding to the single utteranceof the predefined hotword by the user and without requiring the user tospeak additional utterances of the predefined hotword.
 8. The computerimplemented method of claim 2, wherein the hotword detection model isbased at least on the single utterance and not based on anotherutterance of the predefined hotword.
 9. A system comprising: one or morecomputers and one or more storage devices storing instructions that areoperable, when executed by the one or more computers, to cause the oneor more computers to perform operations comprising: receiving audio datacorresponding to a single utterance by a user of a predefined hotword,wherein the predefined hotword is pronounced by the user using apersonalized, non-standard pronunciation; in response to receiving theaudio data corresponding to the single utterance by the user of thepredefined hotword, downloading audio features corresponding to otherusers' utterances of the same, predefined hotword in a manner that isindicated as similar to the personalized, non-standard pronunciation;dynamically generating a hotword detection model for the personalized,non-standard pronunciation of the predefined hotword using (i) the audiodata corresponding to the single utterance by the user of the predefinedhotword, and (ii) the downloaded audio features corresponding to otherusers' utterances of the same, predefined hotword; and using thedynamically generated hotword detection model to detect a likelyutterance of the hotword in subsequently received audio data.
 10. Thesystem of claim 9, the operations further comprise: during an enrollmentprocess, prompting, by a client device, the user to speak the predefinedhotword; and generating enrollment acoustic data using the receivedaudio data from the user, wherein the audio data comprises thepredefined hotword pronounced by the user using the personalized,non-standard pronunciation and additional one or more terms spoken bythe user that trigger semantic interpretation of the one or more termsthat follow the predefined hotword.
 11. The system of claim 10, theoperations further comprise: obtaining a set of candidate acoustic datarepresenting utterances that were previously-spoken by the other users,wherein the other users are of a similar type of user to the user. 12.The system of claim 11, wherein obtaining the set of candidate acousticdata representing the utterances that were spoken by the other users theoperations further comprise: determining, for each candidate acousticdata of the set of candidate acoustic data, a similarity score thatrepresents an acoustic similarity between the enrollment acoustic dataand the candidate acoustic data.
 13. The system of claim 12, whereindetermining the similarity score that represents an acoustic similaritybetween the enrollment acoustic data and the candidate acoustic data theoperations further comprise: determining a plurality of sub-similarityscores between the enrollment acoustic data and the candidate acousticdata; and determining the similarity score based on an averaging of theplurality of sub-similarity scores.
 14. The system of claim 9, whereindynamically generating the hotword detection model for the personalized,non-standard pronunciation of the predefined hotword using (i) the audiodata corresponding to the single utterance by the user of the predefinedhotword, and (ii) the downloaded audio features corresponding to otherusers' utterances of the same, predefined hotword the operations furthercomprise: training the hotword detection model to detect the likelyutterance of the predefined hotword by the user in the subsequentlyreceived audio data corresponding to the single utterance of thepredefined hotword by the user and without requiring the user to speakadditional utterances of the predefined hotword.
 15. The system of claim9, wherein the hotword detection model is based at least on the singleutterance and not based on another utterance of the predefined hotword.16. A non-transitory computer-readable medium storing softwarecomprising instructions executable by one or more computers which, uponsuch execution, cause the one or more computers to perform operationscomprising: one or more computers and one or more storage devicesstoring instructions that are operable, when executed by the one or morecomputers, to cause the one or more computers to perform operationscomprising: receiving audio data corresponding to a single utterance bya user of a predefined hotword, wherein the predefined hotword ispronounced by the user using a personalized, non-standard pronunciation;in response to receiving the audio data corresponding to the singleutterance by the user of the predefined hotword, downloading audiofeatures corresponding to other users' utterances of the same,predefined hotword in a manner that is indicated as similar to thepersonalized, non-standard pronunciation; dynamically generating ahotword detection model for the personalized, non-standard pronunciationof the predefined hotword using (i) the audio data corresponding to thesingle utterance by the user of the predefined hotword, and (ii) thedownloaded audio features corresponding to other users' utterances ofthe same, predefined hotword; and using the dynamically generatedhotword detection model to detect a likely utterance of the hotword insubsequently received audio data.
 17. The computer-readable medium ofclaim 16, the operations comprising: during an enrollment process,prompting, by a client device, the user to speak the predefined hotword;and generating enrollment acoustic data using the received audio datafrom the user, wherein the audio data comprises the predefined hotwordpronounced by the user using the personalized, non-standardpronunciation and additional one or more terms spoken by the user thattrigger semantic interpretation of the one or more terms that follow thepredefined hotword.
 18. The computer-readable medium of claim 17, theoperations comprising: obtaining a set of candidate acoustic datarepresenting utterances that were previously-spoken by the other users,wherein the other users are of a similar type of user to the user. 19.The computer-readable medium of claim 18, wherein obtaining the set ofcandidate acoustic data representing the utterances that were spoken bythe other users the operations comprising: determining, for eachcandidate acoustic data of the set of candidate acoustic data, asimilarity score that represents an acoustic similarity between theenrollment acoustic data and the candidate acoustic data.
 20. Thecomputer-readable medium of claim 19, wherein determining the similarityscore that represents an acoustic similarity between the enrollmentacoustic data and the candidate acoustic data the operations comprising:determining a plurality of sub-similarity scores between the enrollmentacoustic data and the candidate acoustic data; and determining thesimilarity score based on an averaging of the plurality ofsub-similarity scores.
 21. The computer-readable medium of claim 16,wherein dynamically generating the hotword detection model for thepersonalized, non-standard pronunciation of the predefined hotword using(i) the audio data corresponding to the single utterance by the user ofthe predefined hotword, and (ii) the downloaded audio featurescorresponding to other users' utterances of the same, predefined hotwordthe operations comprising: training the hotword detection model todetect the likely utterance of the predefined hotword by the user in thesubsequently received audio data corresponding to the single utteranceof the predefined hotword by the user and without requiring the user tospeak additional utterances of the predefined hotword.